A rocky journey toward effective assessment of visualization modules for learning enhancement in Engineering Mechanics

Beginning in the fall of 1998 and continuing to the present, an assessment study has been in progress to determine the effectiveness of visualization modules used in a basic engineering class at the United States Air Force Academy (USAFA). This paper discusses the progression of this assessment study. Until recently, there has been a lack of content designed to enhance understanding of basic engineering mechanics through the use of visualization. Therefore, visualization content in this area, as well as quantitative assessment establishing its effectiveness, has been needed. The visualization content used in this study consists of web-based and PowerPoint presentations designed to enhance understanding of abstract concepts in the course. Various assessment techniques have been used to evaluate the visual content’s effectiveness. The 1998 version of the study attempted to correlate too many variables resulting predominately in data which was statistically insignificant. Both the visualization modules and the assessment plan were refined based on what was learned in the 1998 study. In 1999, three professors at USAFA ran simultaneous studies using the refined version of the visualization modules. The assessment results from this study produced two interesting results: 1) the attitude of the professor presenting the visualization module can have a significant impact on the student’s reception of the content and 2) students actually disliked the use of the visualization modules. Based on the data from this study, the visual material and the assessment plan were again reworked. Based on additional 1999 assessment data, the hypothesis was formulated that the students’ negative perception was based on three things: 1) they were not aware that this visual content would help them prepare for upcoming exams, 2) the parts of the visual content that gave an overview of an advanced engineering analysis technique called finite elements were intimidating and 3) they appeared to be influenced by one professor’s negative perception of the visualization modules. In order to test this hypothesis, in fall 2000 the visual content was reused, but the link between this content and the conceptual questions on the exam was emphasized. In addition, the non-essential content related to finite element analysis was removed and the professor who had a negative perception of the modules chose not to participate in the study. The latest assessment results indicate the students’ perception of the material has improved significantly in response to these changes. In addition, the fall 2000 assessment shows that the visual modules did enhance understanding when compared to a traditional lecture format. Overall, this three-year assessment project should provide others developing visualization and other content with important information relevant to the assessment processes.